Optimal objective function in high-dimensional regression

In this article we study a fundamental statistical problem: how to optimally pick the objective to be minimized in a parametric regression when we have information about the error distribution. The classical answer to the problem we posed at the beginning, maximum likelihood, was given by Fisher (5) in the specific case of multinomial models and then at succeeding levels of generality by Cramér (3), Hájek (7) and above all Le Cam (11). For instance, for p fixed or p/n → 0 fast enough, least squares is optimal for Gaussian errors while LAD is optimal for double exponential errors. We shall show that this is no longer true in the regime we consider with the answer depending, in general, on the limit of the ratio p/n as well as the form of the error distribution. Our analysis in this paper is carried out in the setting of Gaussian predictors, though as we explain below, this assumption should be relaxable to a situation where the distribution of the predictors satisfy certain concentration properties for quadratic forms. We carry out our analysis in a regime which has been essentially unexplored, namely 0 p/n < 1 where p is the number of predictor variables and n is the number of independent observations. Since in most fields of application, situations where p as well as n is large have become paramount, there has been a huge amount of literature on the case where p/n 0 but the number of “relevant” predictors is small. In this case the objective function, quadratic (least squares) or otherwise (`1 for LAD) has been modified to include a penalty (usually `1) on the regression coefficients which forces sparsity ((1)). The price paid for this modification is that estimates of individual coefficients are seriously biased and statistical inference, as opposed to prediction, often becomes problematic. In (4), we showed 1 that this price need not be paid if p/n stays bounded away from 1. We review the main theoretical results from this previous paper in Result 1 below. From a practical standpoint, some of our key findings were: